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CROSSMODAL  CONGRUENCY  BENEFITS  FOR  TACTILE  AND  VISUAL  SIGNALING 

James  L.  Merlo  Richard  Gilson  P.  A.  Hancock 

United  States  Military  Academy  University  of  Central  Florida  University  of  Central  Florida 

West  Point,  NY  10996  Orlando,  FL  32826  Orlando,  FL  32826 

An  experiment  is  reported  in  which  tactile  messages  were  created  based  on  five  common  military  arm  and 
hand  signals.  We  compared  response  times  and  accuracy  rates  of  novice  individuals  responding  to  visual  and 
tactile  representations  of  these  messages.  Such  messages  were  displayed  either  alone  or  in  congruent  or 
incongruent  combinations.  Analyses  were  conducted  on  trials  where  tactile  and  visual  signals  messages  were 
presented  either  individually  or  concurrently.  Results  indicated  a  beneficial  effect  for  congruent  message 
presentations  with  both  modalities  showing  a  superior,  combined  response  time  and  improved  accuracy  when 
compared  to  individual  presentations  in  either  modality  alone.  These  results  confirm  the  promise  for  tactile 
messages  to  augment  visual  displays  in  challenging  and  stressful  environments  where  visual  messaging  may 
not  always  be  clear  or  even  possible. 


Introduction 

Many  operational  conditions  such  as  combat,  fire 
fighting,  or  certain  law  enforcement  and/or  emergency 
management  situations  impose  significant  demands  on 
operator’s  sensory  capabilities.  Noisy  (e.g.,  weapons 
fire,  vehicle  engines,  etc.)  and  murky  (e.g.,  smoke, 
sandstorms)  environments,  for  example,  impose  great 
demands  on  hearing  and  vision,  and  can  compromise 
the  ability  to  exchange  critical  information  through 
conventional  communication  pathways  (Merlo, 
Szalma,  &  Hancock,  2007).  To  circumvent  these 
environmental  barriers  it  may  be  possible  to  provide  a 
redundant  source  of  information  through  the  modality 
of  touch,  by  using  tactile  signaling. 

Previous  studies  have  shown  tactile  systems  can 
produce  relatively  stable  performance  improvements 
across  a  variety  of  body  orientations  even  when 
spatial  translation  is  required  (Oron-Gilad,  Downs, 
Gilson,  &  Hancock,  2007;  Terrence,  Brill,  &  Gilson, 
2005)  as  well  as  in  the  presence  of  physiological 
stress  (Merlo,  Stafford,  Gilson,  &  Hancock,  2006). 
Comprehensive  review  of  these  and  other  tactile 
studies  have  recently  appeared  (see,  Gilson,  Redden, 
&  Elliot,  2007;  Prewett,  Yang,  Stilson,  Gray,  Coovert, 
&  Burke,  2006). 

Most  of  human  information  processing  uses 
multiple  sensory  inputs,  primarily  involving  the 
synthesis  of  visual  and  auditory  cues  (Hancock,  2005; 
Spence  &  Driver,  2004;  Stein  &  Meredith,  1993). 
Literature  on  experiments  that  involve  the  use  of  two 
modalities  of  information  presented  redundantly, 
namely  tactile  and  visual,  each  show  improvement  in 
the  areas  of  accuracy  and  response  time  (Spence  & 
Walton,  2005;  Gray  &  Tan,  2002;  Strybel  &  Vatakis, 
2004).  The  present  study  seeks  to  show  that  similar 


congruency  benefits  may  be  achieved  for  more 
complex  stimuli  presented  through  both  the  visual  and 
tactile  modalities. 

Experimental  Method 

Experimental  Participants 

To  investigate  the  foregoing  proposition,  twenty 
participants  (9  males  and  1 1  females)  ranging  in  age 
from  18  to  48,  with  an  average  age  of  25  years, 
volunteered  to  participate.  Each  participant  self- 
reported  no  surgeries,  significant  scarring  or  any 
impediment  that  might  cause  lack  of  feeling  in  the 
abdomen  or  torso  area.  None  of  the  participants  had 
any  prior  experience  with  the  presented  arm  and  hand 
signals  nor  the  tactile  signals  in  general,  beyond  the 
possible  familiarity  with  cell-phone  vibratory  alerts. 

Experimental  Materials  and  Apparatus 

The  vibrotactile  actuators  (tactors)  in  our  system 
are  model  C2,  manufactured  by  Engineering 
Acoustics,  Inc.  They  are  acoustic  transducers  that 
displace  200-300  Hz  sinusoidal  vibrations  onto  the 
skin.  Their  17  gm  mass  is  sufficient  for  activating  the 
skin’s  tactile  receptors.  The  C2’s  contactor  is  7  mm, 
with  a  1  mm  gap  separating  it  from  the  tactor 
aluminum  housing.  The  C2  is  a  tuned  device,  meaning 
it  operates  well  only  within  a  very  restricted 
frequency  range,  in  this  case  approximately  250  Hz. 
The  tactile  display  itself  is  a  belt  like  device  with  eight 
vibrotactile  actuators,  an  example  of  which  is  shown 
in  Figure  1.  The  belt  itself  is  made  of  elastic  and  high 
quality  cloth  similar  to  the  material  used  by 
professional  cyclists.  When  stretched  around  the  body 
and  fastened,  the  wearer  has  an  actuator  over  the 
umbilicus  and  one  centered  over  his  spine  in  the  back. 
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The  other  six  actuators  are  equally  spaced,  three  on 
each  side,  for  a  total  of  eight  (see  Cholewiak,  Brill,  & 
Schwab,  2004). 


Figure  1.  Three  tactile  displays  belt  assemblies  are 
shown  above  along  with  their  controller  box. 


The  tactors  are  operated  using  a  Tactor 
Control  Unit  (TCU)  which  is  a  computer-controlled 
driver/amplifier  system  that  switches  each  tactor  on 
and  off  as  required.  This  device  is  shown  on  the  left 
side  of  the  tactile  displays  belts  in  Figure  1.  The 
TCU  weighs  1.2  lbs  independent  of  its  power  source 
and  is  approximately  one  inch  thick.  This  device 
connects  to  a  power  source  with  one  cable  and  to  the 
display  belt  with  the  other  and  uses  Bluetooth 
technology  to  communicate  with  the  computer 
driven  interface.  Tactile  messages  were  created 
using  five  standard  Army  and  Marine  Corps  arm  and 
hand  signals  (Department  of  the  Army,  1987).  The 
five  signals  chosen  for  the  experiment  were, 
“Attention”,  “Halt”,  “Rally”,  “Move  Out”,  and 
“Nuclear  Biological  Chemical  event  (NBC)”.  The 
tactile  representations  of  these  signals  were  designed 
in  a  collaborative  effort  of  scientists  at  the 
University  of  Central  Florida  and  a  consultant  group 
of  subject  matter  experts  (SMEs)  consisting  of 
former  US  Soldiers  and  Marines. 

Short  video  clips  of  a  soldier  performing  the 
five  arm  and  hand  signals  were  edited  to  create  the 
visual  stimuli.  Careful  editing  ensured  the  timing  of 
the  arm  and  hand  signals  closely  matched  that  of  the 
tactile  presentations  (see  Figure  2).  A  Samsung  Q1 
Ultra  Mobile  computer  using  an  Intel  Celeron  M 
ULV  (900  MHz)  processor  with  a  7”  WVGA  (800  x 
480)  liquid  crystal  display  was  used  to  present 
videos  of  the  soldier  performing  the  arm  and  hand 
signals.  This  computer  ran  a  custom  Lab  VIEW  (8.2; 
National  Instruments)  application  that  presented  the 
tactile  signals  via  Bluetooth  to  the  tactor  controller 
board  and  captured  all  of  the  participant’s  responses 
via  mouse  input.  Participants  wore  sound  dampening 
headphones  with  a  reduction  rating  of  11.3  dB  at 


250  Hz  to  reduce  the  effects  of  any  auditory  stimuli 
emanated  by  the  tactor  actuation. 

The  display  of  each  message  or  signal  was  presented 
in  one  of  four  ways: 

•  visual  only  (video  presentation  of  the  arm  and 
hand  signal) 

•  tactile  only  (tactile  presentation  of  the  arm  and 
hand  signal) 

•  both  visual  and  tactile  simultaneous  and 
congruent  (i.e.  the  same  signals  were  presented 
both  through  the  video  and  through  the  tactile 
system) 

•  Both  visual  and  tactile  simultaneous  and 
incongruent  (i.e.  the  visually  presented  signal  did 
not  match  the  presented  tactile  signal) 


Lik  BJmkm 


Figure  2.  A  computer  screen  shot  showing  what  the 
participant  viewed  as  the  signals  were  presented.  The 
participant  mouse  clicked  on  the  appropriate  signal  name 
after  each  presentation. 

Experimental  Design  and  Procedure 

Participants  first  completed  a  computer- 
based  tutorial  that  described  each  arm  and  hand 
signal  individually.  For  each  signal,  a  short 
description  was  presented.  Participants  then  viewed 
a  video  of  a  soldier  performing  the  signal  followed 
by  the  experience  of  its  tactile  equivalent.  Finally, 
the  participants  were  able  to  play  the  signals 
concurrently  (both  visual  and  tactile  representation 
at  the  same  time).  Participants  were  allowed  to 
repeat  the  presentation  (i.e.,  visual,  tactile,  visual- 
tactile)  as  many  times  as  desired.  Once  the 
participant  reviewed  the  five  signals  in  the  two 
presentation  styles,  a  validation  exercise  was 
performed.  Participants  had  to  correctly  identify 
each  signal  twice  before  the  computer  would  prompt 
the  experimenter  that  the  participant  was  ready  to 
begin. 

Each  participant  performed  two,  sixty  trial 
blocks.  The  blocks  had  two  of  each  signal  presented 
only  visually  (10  total),  two  of  each  signal  with  only 
tactile  signals  (10  total),  four  of  each  signal 
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performed  simultaneously  with  both  congruent 
visual  and  tactile  presentation  (20)  and  four  of  each 
tactile  and  visual  signal  performed  simultaneously 
but  as  incongruent  presentations  (20).  Each 
participant  performed  two  blocks  of  trials,  with  the 
sixty  trials  within  the  blocks  completely  randomized 
for  each  participant.  For  any  one  indivisual 
participant,  the  entire  procedure  took  less  than  an 
hour  to  complete. 

Results 

All  analyses  reported  were  conducted  using 
SPSS  11.5  for  Windows  with  the  alpha  level  set  at 
.05  for  a  two-tailed  t-test  conducted  unless  otherwise 
noted.  Results  were  analyzed  in  terms  of  the  speed 
of  response  and  the  accuracy  of  that  response  under 
the  respective  conditions.  Only  the  results  from  the 
three  conditions  of  visual  presentation,  tactile 
presentation  and  congruent,  concurrent  presentation 
are  presented  in  the  present  paper. 

A  one-way  Analysis  of  Variance  (ANOVA) 
was  performed  on  the  mean  response  times  across 
the  three  experimental  conditions  of  visual 
presentation,  tactile  presentation  or  visual-tactile 
concurrent  and  congruent  presentation,  with  the 
following  result:  F(  1,  19)=473.45,  p<. 01,  (r|2p  = 
.961,(3=  1.00). 


Subsequent  a  priori  pairwise  analysis  showed, 
simultaneously  presented  congruent  signals  resulted 
in  significantly  faster  response  times  than  visual 
signals  presented  alone  £(19)=-2.25,  p<. 04,  see 
Figure  3  below.  Also,  as  is  evident,  the  congruent 
signals  were  faster  than  tactile  alone  ^(19)=-3.98, 
p<. 01.  Additionally,  the  visual  only  presentation  of 
the  signal  was  significantly  faster  than  the  tactile 
only  presentation  of  the  signal  t(19)=-2.16,/?<.04. 

Although,  there  was  no  traditional  level  of 
significant  difference  in  accuracy  rates  observed 
between  the  visual  and  tactile  signals  when 
presented  alone  t{  19)=2.00,  p<. 06,  there  was  a 
significant  difference  in  the  performance  rates  when 
the  tactile  modality  was  compared  to  the  concurrent 
congruent  presentation  of  the  signals,  £(19)=4.03, 
p<. 01.  The  overall  lower  accuracy  rate  for  the  tactile 
signaling  was  due  to  an  apparent  confusion  between 
the  tactile  signal  for  ‘NBC”  and  ‘Halt”,  which  have 
similar  tactile  characteristics  but  comparatively  low 
visual  similarity.  Analysis  without  the  “NBC”  tactile 
signal  data  removed  these  differences  (it  was  not 
significantly  different  with  it  included)  in  the  error 
rate  between  visual  and  tactile  signals,  while  not 
influencing  the  main  effect  between  congruent 
modality  signaling  and  single  modality  signaling  at 
all. 


Figure  3.  Response  time  in  milliseconds  by  signal  presentation  condition. 
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Discussion 

The  overall  high  accuracy  rate  displayed  by  the 
participants  (over  80%  across  all  modalities  with 
fewer  than  ten  minutes  of  training)  is  highly 
encouraging  to  the  current  form  of  tactile  display.  The 
accuracy  of  the  messages  and  the  reported 
intuitiveness  with  which  they  were  received  is  also  a 
testament  to  the  utility  of  the  subject  matter  expert 
information  and  the  present  tactile  ‘language  ’ 
transformation  format.  Similarities  in  both  tactile  and 
visual  signals  that  cause  confusion  among  signals  are 
virtually  eliminated  in  concurrent  presentation.  The 
rich  multi-modal  information  format  seemed  to 
produce  faster  and  more  accurate  performance. 

The  question  of  learning  complex  tactile 
communication  signals,  especially  for  use  in  adverse 
or  unusual  circumstances  is  liable  to  be  an  important 
future  issue.  The  tactile  system  acts  as  a  redundancy 
gain  as  the  participants  now  have  two  means  of 
receiving  communication  because  the  visual  hand  and 
arm  signals  would  still  be  available.  While  initial 
testing  seems  to  result  in  superior  performance  for 
tactile  communication  and  traditional  arm  and  hand 
signals  combined,  the  challenge  of  a  universal  input 
device  remains  a  significant  hurdle.  Stimulus  response 
compatibility  will  have  to  be  analyzed  carefully  to 
maximize  performance  as  different  types  of  inputs  are 
considered  for  use  with  tactile  displays.  However, 
when  individuals  are  faced  with  extreme  challenges 
and  the  traditional  sources  of  information  are  either 
diminished  of  eliminated  altogether,  the  tactile  system 
provides  an  important  alternative  communication 
channel  and  one  that  can  and  should  be  exploited. 

Up  until  the  present  innovative  development 
in  tactor  technology  ,  the  results  for  tactile  signaling 
alone,  or  in  combination  with  other  modalities,  has 
been  rather  mixed.  Now  that  clear  and  unequivocal 
messages  can  be  supplied  to  various  body  locations 
including  most  interestingly,  the  tongue,  the 
opportunity  to  exploit  this  relatively  unexploited 
sensory  channel  presents  itself.  However,  beyond 
the  practical  and  pragmatic  issues  of  display 
generation,  these  issues  of  multi-sensory  signaling 
pose  critical  theoretical  challenges  as  to  the  temporal 
sequencing  of  stimulation  across  the  information 
processing  system.  We  will  have  to  know  much 
more  about  the  buffering  and  prioritizations  of 
multiple  competing  cues  in  order  to  avoid  confusing 
or  contradictory  communications.  We  will  have  to 
explore  the  nature  of  the  primacy  of  “direct”  versus 
“interpretive”  tactile  signals  and  especially  what 


connotes  “intuitively  obvious”  signals  in  this  realm. 
Although  these  represent  significant  challenges, 
these  early  results  suggest  that  the  practical  benefits 
alone  which  can  accrue  will  be  worth  the  effort. 
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